Statistical Analysis and Application of Ensemble Method on the Netflix Challenge

1. Introduction The Netflix Prize project is proposed by the Neflix Inc., in order to seek accurate predictions on movie ratings. As one group in the Stanford Netflix Prize team, our responsibility is to explore useful statistics and data curation in the training data set, and to explore ensemble methods for improving prediction accuracies. We imported the Netflix data into a MySQL database for data aggregation, and then the aggregated results can be analyzed using Matlab or C++ scripts. So far, we have finished multiple clustering analyses to the movies and the customers by the K-means clustering techniques learnt from class [1]. We clustered the movies by multiple interesting criteria, such as the number of ratings to a movie, the average ratings to a movie, time progression on monthly numbers of ratings and rating averages, and the probability of different ratings for a movie. The customers are clustered with similar criteria except the time progression because the monthly numbers of ratings and rating averages change from time to time, depending on the movies the customers watch in those months. After the training data have been properly clustered through various criteria, we used ensemble methods to effectively combine the advantages of various classifiers and obtain improved results.